This section is intended to introduce the reader to various aspects of art, which may be related to various aspects of the present invention that are described and/or claimed below. This discussion is believed to be helpful in providing the reader with background information to facilitate a better understanding of the various aspects of the present invention. Accordingly, it should be understood that these statements are to be read in this light, and not as admissions of prior art.
The acquisition of 4D light-field data), which can be viewed as a sampling of a 4D light field (i.e. the recording of light rays as explained in FIG. 1 of the article: “Understanding camera trade-offs through a Bayesian analysis of light field projections” by Anat Levin et al., published in the conference proceedings of ECCV 2008) is an hectic research subject.
Indeed, compared to classical 2D images obtained from a camera, 4D light-field data enable a user to have access to more post processing features that enhance the rendering of images and/or the interactivity with the user. For example, with 4D light-field data, it is possible to perform with ease refocusing of images a posteriori (i.e. refocusing with freely selected distances of focalization meaning that the position of a focal plane can be specified/selected a posteriori), as well as changing slightly the point of view in the scene of an image. In order to acquire 4D light-field data, several techniques can be used. Especially, a plenoptic camera, as depicted in document WO 2013/180192 or in document GB 2488905, is able to acquire 4D light-field data. Details of the architecture of a plenoptic camera are provided in FIGS. 1, 2, 3, 4 and 5 of the present document.
In the state of the art, there are several ways to represent (or define) 4D light-field data. Indeed, in the Chapter 3.3 of the Phd dissertation thesis entitled “Digital Light Field Photography” by Ren Ng, published in July 2006, three different ways to represent 4D light-field data are described. Firstly, 4D light-field data can be represented, when recorded by a plenoptic camera as the one depicted in FIG. 1 for example, by a collection of micro-lens images (see the description of FIG. 2 in the present document). 4D light-field data in this representation are named raw images (or 4D raw light-field data). Secondly, 4D light-field data can be represented, by a set of sub-aperture images. A sub-aperture image corresponds to a captured image of a scene from a point of view, the point of view being slightly different between two sub-aperture images. These sub-aperture images give information about the parallax and depth of the imaged scene. Thirdly, 4D light-field data can be represented by a set of epipolar images (see for example the article entitled: “Generating EPI Representation of a 4D Light Fields with a Single Lens Focused Plenoptic Camera”, by S. Wanner et al., published in the conference proceedings of ISVC 2011).
A common technique that is used to perform refocusing from 4D light-field data is based on the shift and addition of micro-lens images (i.e. directly from the 4D raw light-field data, note IRAW) as explained in document WO 2013/167758 (in the section “image refocusing method”). The FIG. 6 of the present document roughly depicts how to project a pixel at coordinates (x, y) in said 4D raw light-field data for obtaining a 2D image. While 4D raw light-field pixels (i.e. the pixels of the micro-images) are projected into a refocused image, a weight-map records the number of accumulated projected pixels. The weight-map also records the interpolation in the case that projected coordinates are non-integer coordinates. Once all 4D raw light-field pixels are projected into the refocused image and the weight-map is determined, the refocused image is divided by the weight-map so that each refocused pixel received the same average contribution. The resulting 2D image can be output on a display, or stored and/or transmitted to another device for example.
In order to improve the quality (especially the sharpness) of the 2D refocused image, the technique described in the article entitled “Refocusing Plenoptic Images using Depth-Adaptive Splatting” by Juliet Fiss et al. can be used for interpolation purpose. Indeed, the interpolation based on this approach consists in spreading the influence of a 4D raw light-field pixel on the 2D refocused image (see the FIG. 5 of the article where a 4D raw light-field pixel at coordinates (x, y) is projected to a location s with a value defined as a function of a splat kernel). As mentioned in this article: “Splatting can be viewed as a form of scattered data interpolation using radial basis functions”. However, one drawback of this approach is that 4D raw light-field data have to be demosaicked before projecting the 4D raw light-field pixels onto the 2D refocused image. Another drawback of this approach is that the splat kernel is isotropic in (x, y) and only depends on the depth of the scene. At last, another drawback of this approach is that the splat kernel does not take into account multi-focal plenoptic cameras (as for example the one depicted in the article entitled “The Multi-Focus Plenoptic Camera” by Todor Georgiev and Andrew Lumsdaine) or the geometry of the scene.
Therefore, there is a need to provide a technique that can overcome these drawbacks.